Simple Linear Regression¶

This is a simple linear regression model as in every econometrics textbooks $$ Y_i=\beta_1+\beta_2X_i+u_i $$ where $Y$ is dependent variable, $X$ is independent variable and $u$ is disturbance term. $\beta_1$ and $\beta_2$ are unknown parameters that we are aiming to estimate by feeding the data in the model. Without disturbance term, the model is simple a function of a straight line in $\mathbb{R}^2$, such as $$ Y = 2 + 3X $$

In the context of machine learning (ML), the $X$ is usually called feature variable and $Y$ called target variable. And linear regression is the main tool in supervised learning, meaning that $Y$ is supervising $X$./

There are five reasons justified that we need a disturbance term:

1. omission of independent variables
2. aggregation of variables
3. model misspecification
4. function misspecification, eg. should be nonlinear rather than linear
5. measurement error

The second one means that if we intend to aggregate the variable to a macro level, for instance every family has a consumption function, but aggregation on a national level causes discrepancies which contribute to the disturbance term.

The third and forth one will be discussed in details in later chapter.

The fifth one includes all types of error, man-made or natural.

Odinary Least Squares¶

Odinary Least Squares is the most common estimation technique used in ML or econometrics, it is popular due to its simplicity and transparency. You'll be able to derive the whole estimation process by hand-calculation, all steps will have closed-form expression.

We'll demonstrate OLS with our first plot. Every time you run this script, the result will be different than mine, because no random seeds are set.

We have plotted a fitted line onto $10$ observations (red dots) which was generated by $Y_i = 2+3X_i+5u_i$, where $u_i \sim N(0, 1)$. For easy demonstration, say we have a 'perfect' estimator that provides $$ \hat{\beta}_1 = 2\\ \hat{\beta}_2 = 3 $$ where $\hat{\beta}_1 $ and $\hat{\beta}_2$ are estimates, in contrast $\beta_1$ and $\beta_2$ are model parameters.

Therefore we can plot a fitted line (blue line) $\hat{Y} = 2+3X$. The red dashed line is the difference of $Y_i$ and $\hat{Y}_i$, we officially call it residual, denoted as $\varepsilon_i$. $$ \varepsilon_i = Y_i - \hat{Y}_i = Y_i - \hat{\beta}_1-\hat{\beta}_2X_i $$ The OLS algorithm is aiming find the estimates of $\beta_1$ and $\beta_2$ such that $$ \text{min}\text{ RSS}=\sum_{i=1}^n \varepsilon^2_i $$ where $RSS$ is the residual sum of squares $(\text{RSS})$.

In ML context, notation is slightly different, but the ideas are the same, a cost function is defined with $\text{RSS}$ $$ J(\hat{\beta_1}, \hat{\beta}_2) = \frac{1}{2m} \sum_{i=1}^n \varepsilon^2_i $$ Then minimize the cost function $$ \text{min }J(\hat{\beta_1}, \hat{\beta}_2) $$

To minimise the $RSS$, we simply take partial derivatives w.r.t. $b_2$ and $b_1$ respectively. The results are the OLS estimators of simple linear regression, which are \begin{align} \hat{\beta}_2 &=\frac{\sum_{i=1}^n(X_i-\bar{X})(Y_i-\bar{Y})}{\sum^n_{i=1}(X_i-\bar{X})^2}=\frac{\text{Cov}(X, Y)}{\text{Var}(X)}\\ \hat{\beta}_1 &= \bar{Y}-\hat{\beta}_2\bar{X} \end{align}

With these formulae in mind, let's perform a serious OLS estimation. Considering possible repetitive use of OLS in this tutorial, we will write a class for OLS.

A convenient function np.polyfit of curve fitting could verify our results.

The plot the fitted line $b_1+b_2X$, original line $\beta_1+\beta_2X$ and observations.

From the estimation result and graph above, we can notice $\hat{\beta}_1$ and $\hat{\beta}_2$ are close to true parameters $\beta_1$ and $\beta_2$ if scalar of disturbance term is small.

Interpretation of Estimated Coefficients¶

We will check on a real data set to understand basic principles of interpreting estimates. The data set collected average apartment price ¥/$m^2$ and average annual disposable income from $25$ Chinese cities in 2020.

Load the data with Pandas.

Estimate parameters with OLS algorithm.

Plot the observation and regression line.

$\hat{\beta}_2$ can be interpreted literally as the graph shows, as disposable income increases $1$ yuan (Chinese currency unit), the house price increases $1.1$ yuan.

As for $\hat{\beta}_1$, it is tricky to interpret the face value that if the disposable income is zero, i.e. the house price is $-29181$ yuan if disposable income is $0$, it doesn't make any sense. The basic principle of interpreting constant term is to check if it has a plausible meaning when all independent/feature variables equals zero. If no sensible meaning, you don't need to interpret it, this is a well known defect of linear regression model.

Important Results of OLS¶

Some features of OLS could provide us insights of mechanism of the algorithm.

The first one is $$ \bar{\varepsilon}=0 $$ It is true because $$ \bar{\varepsilon}=\bar{Y}-\hat{\beta}_1-\hat{\beta}_2\bar{X}=\bar{Y}-(\bar{Y}-\hat{\beta}_2\bar{X})-\hat{\beta}_2\bar{X}=0 $$ holds. We can demonstrate numerically with the variables that we have defined in house price example.

It is not theoretically zero due to some numerical round-off errors, but we treat it as zero.

The second feature is $$ \bar{\hat{Y}}=\bar{Y} $$

The third and forth features are $$ \sum_i^n X_i\varepsilon_i=0\\ \sum_i^n \hat{Y}_i\varepsilon_i=0 $$

This can be shown by using a dot product function.

Actually, lots of econometric theory can be conveniently derived by linear algebra.

For instance, in linear algebra, covariance has a geometric interpretation $$ \text{Cov}(X, Y)=x\cdot y= ||x||||y||\cos{\theta} $$ where $x$ and $y$ are vectors in $\mathbb{R}^n$. If dot product equals zero, geometrically these two vectors are perpendicular, denote as $x\perp y$. Therefore the third and forth features are equivalent to $$ \text{Cov}(X, e)=0\\ \text{Cov}(\hat{Y}, e)=0 $$ i.e. $x\perp e$ and $\hat{y} \perp e$. Traditionally, the vectors are denoted as lower case letters.

Variance of Decomposition¶

Variance of decomposition is based on analysis of variance (ANOVA), if you don't know what ANOVA is, check here. We know that any observation can be decomposed as a fitted value and a residual $$ Y_i = \hat{Y}_i+\varepsilon_i $$ Take variance on both sides $$ \text{Var}(Y)=\text{Var}(\hat{Y}+\varepsilon)=\operatorname{Var}(\hat{Y})+\operatorname{Var}(\varepsilon)+ \underbrace{2 \operatorname{Cov}(\hat{Y}, \varepsilon)}_{=0} $$ Or in the explicit form $$ \frac{1}{n} \sum_{i=1}^{n}\left(Y_{i}-\bar{Y}\right)^{2}=\frac{1}{n} \sum_{i=1}^{n}\left(\hat{Y}_{i}-\overline{\hat{Y}}\right)^{2}+\frac{1}{n} \sum_{i=1}^{n}\left(\varepsilon_{i}-\bar{\varepsilon}\right)^{2} $$

Use the OLS features, i.e. $\bar{\hat{Y}}=\bar{Y}$ and $\bar{\varepsilon}=0$, the equation simplifies into $$ \underbrace{\sum_{i=1}^{n}\left(Y_{i}-\bar{Y}\right)^{2}}_{TSS}=\underbrace{\sum_{i=1}^{n}\left(\hat{Y}_{i}-\bar{Y}\right)^{2}}_{ESS}+\underbrace{\sum_{i=1}^{n} \varepsilon_{i}^{2}}_{RSS} $$ where $TSS$ means total sum of squares, $ESS$ means explained sum of squares.

Coefficient of Determination¶

Though $ESS$ is called 'explained' part, it might be entirely wrong due to misspecification of model. That being said, we still need a quantitative indicator that tells us how much the model is able to 'explain' the behaviour of the dependent variables.

The coefficient of determination is most intuitive indicator $$ R^2 = \frac{ESS}{TSS}=\frac{\sum_{i=1}^{n}\left(\hat{Y}_{i}-\bar{Y}\right)^{2}}{\sum_{i=1}^{n}\left(Y_{i}-\bar{Y}\right)^{2}} $$

We have written a r_sq() method in the S_OLS class.

It means the disposable income can explain $57\%$ of house price variation. Furthermore, $$ R^2 = \frac{TSS - RSS}{TSS}=1-\frac{RSS}{TSS} $$ it is clear that minimise $RSS$ is equivalent to maximise $R^2$.

Alternatively, the $R^2$ can be shown its relationship with correlation coefficient $r_{Y, \hat{Y}}$.

$$ \begin{aligned} r_{Y, \hat{Y}} &=\frac{\operatorname{Cov}(Y, \hat{Y})}{\sqrt{\operatorname{Var}(Y) \operatorname{Var}(\hat{Y})}}=\frac{\operatorname{Cov}([\hat{Y}+e], \hat{Y})}{\sqrt{\operatorname{Var}(Y) \operatorname{Var}(\hat{Y})}}=\frac{\operatorname{Cov}(\hat{Y}, \hat{Y})+\operatorname{Cov}(e, \hat{Y})}{\sqrt{\operatorname{Var}(Y) \operatorname{Var}(\hat{Y})}}=\frac{\operatorname{Var}(\hat{Y})}{\sqrt{\operatorname{Var}(Y) \operatorname{Var}(\hat{Y})}} \\ &=\frac{\sqrt{\operatorname{Var}(\hat{Y}) \operatorname{Var}(\hat{Y})}}{\sqrt{\operatorname{Var}(Y) \operatorname{Var}(\hat{Y})}}=\sqrt{\frac{\operatorname{Var}(\hat{Y})}{\operatorname{Var}(Y)}}=\sqrt{R^{2}} \end{aligned} $$

Gauss-Markov Conditions¶

In order to achieve the best estimation by OLS, the disturbance term ideally has to satisfy four conditions which are called Gauss-Markov Conditions. Provided that all G-M conditions satisfied, OLS is the preferred over all other estimators, because mathematically it is proved to be the Best Linear Unbiased Estimator (BLUE). This conclusion is called Gauss-Markov Theorem.

1. $E(u_i|X_i)=0$
2. $E(u_i^2|X_i)= \sigma^2$ for all $i$, homoscedasticity
3. $\text{Cov}(u_i, u_j)=0, \quad i\neq j$, no autocorrelation.
4. $\text{Cov}(X_i, u_i)=0$, assuming $X_i$ is non-stochastic.

In addition to G-M conditions, we also assume normality of disturbance term, i.e. $u_i\sim N(0, \sigma^2_u)$, which is guaranteed by Central Limit Theorem.

In practice, almost impossible to have all conditions satisfied simultaneously or even one perfectly, but we must be aware of severity of violations, because any violation of G-M condition will compromise the quality of estimation results. And identifying which condition is violated could also lead us to corresponding remedies.

Random Components of Regression Coefficients¶

According to the OLS formula of $\beta_2$ $$ \hat{\beta}_{2}=\frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)} $$

Plug in $Y=\beta_1+\beta_2X+u$: $$ \hat{\beta}_{2}=\frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)}=\frac{\operatorname{Cov}(X, \beta_1+\beta_2X+u)}{\operatorname{Var}(X)} $$

The covariance operation rules come in handy $$ \operatorname{Cov}(X, \beta_1+\beta_2X+u)=\operatorname{Cov}\left(X, \beta_{1}\right)+\operatorname{Cov}\left(X, \beta_{2} X\right)+\operatorname{Cov}(X, u) $$

where $\operatorname{Cov}\left(X, \beta_{1}\right)=0$, and $\operatorname{Cov}\left(X, \beta_{2} X\right)=\beta_2 \operatorname{Var}(X)$, therefore $$ \hat{\beta}_{2}=\frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)}=\beta_{2}+\frac{\operatorname{Cov}(X, u)}{\operatorname{Var}(X)} $$

If $u$ perfectly uncorrelated with $X$ as in G-M condition, the second term should be $0$. However that rarely happens, so the $\hat{\beta}_2$ and $\beta_2$ will always have certain level of discrepancy. And note that we can't decompose $\hat{\beta}_2$ in practice, because we don't know the true value of $\beta_2$.

And also note that there are two ways to improve the accuracy of $\hat{\beta}_2$, either lower the correlation of $X$ and $u$ or increase the variance of $X$.

Monte Carlo Sampling Distribution¶

We want to perform a Monte Carlo simulation to show the sampling distribution of $\hat{\beta}_1$ and $\hat{\beta}_2$. First, we write a simple class for OLS Monte Carlo experiment.

We can set $\beta_1$, $\beta_2$, $N$ and $a_u$ for initialisation. The model is $$ Y_i=\beta_1+\beta_2X_i +a_uu_i, \qquad i\in(1, n),\qquad u_i\sim N(0,1) $$

Instantiate the OLS Monte Carlo object with $\beta_1=2$, $\beta_2=3$, $n=10$ and $a_u=1$, then run $10000$ times of simulations, each time generated a new $u$. All estimated $\hat{\beta}_1$ and $\hat{\beta}_2$ are collected in their arrays.

Plot the histogram and the mean of estimates. Not difficult to notice that the mean of the $\hat{\beta}_1$ and $\hat{\beta}_2$ are very close to 'true values'.

Now we try again with a larger disturbance scaler $a_u = 2$ and keep rest parameters unchanged.

Though the histogram has been scaled as seemingly identical as above, but pay attention to the $x$-axis and vertical line. Apparently the variance of $u$ affects the accuracy of estimates, i.e. the variance of sampling distribution of $b_1$ and $b_2$.

It's straightforward to see why this happens from the formula $$ \hat{\beta}_{2}=\beta_{2}+\frac{\operatorname{Cov}(X, a_uu)}{\operatorname{Var}(X)}=\beta_{2}+a_u\frac{\operatorname{Cov}(X, u)}{\operatorname{Var}(X)} $$

We know from statistics course the increase sample is always doing good for quality of estimates, thus we dial it up to $N=100$, rest are unchanged.

Statistical Features of Estimates¶

After a visial exam of sample distributions, now we can formally discuss the statistical features of estimator.

Unbiased of Estimator¶

If an estimator is biased, we rarely perform any estimation with it, unless you have no choices which is also rare. But how to prove unbiasness of an estimator?

The rule of thumb is to take expectation on both sides of estimator. Here is the OLS example. $$ E\left(b_{2}\right)=E\left[\beta_{2}+\frac{\operatorname{Cov}(X, u)}{\operatorname{Var}(X)}\right]=\beta_{2}+E\left[\frac{\operatorname{Cov}(X, u)}{\operatorname{Var}(X)}\right]=\beta_2+\frac{E[\operatorname{Cov}(X, u)]}{E[\operatorname{Var}(X)]} $$ To show $E[\operatorname{Cov}(X, u)]=0$, rewrite covariance in explicit form $$ E[\operatorname{Cov}(X, u)]=\frac{1}{n} \sum_{i=1}^{n}\left(X_{i}-\bar{X}\right) E\left(u_{i}-\bar{u}\right)=0 $$ Therefore $$ E\left(\hat{\beta}_{2}\right)=\beta_{2} $$ Again take expectation on $b_1$, immediately we get $$ E(\hat{\beta}_1) = E(\bar{Y})-E(\hat{\beta}_2)E(\bar{X})= E(\bar{Y})-\beta_2\bar{X}=\beta_1 $$ where we used $E(\bar{Y})=\beta_{1}+\beta_{2} \bar{X}$.

Precision of Estimator¶

If we know the variance of disturbance term, the population variance of $b_1$ and $b_2$ can be derived $$ \sigma_{\hat{\beta}_1}^{2}=\frac{\sigma_{u}^{2}}{n}\left[1+\frac{\bar{X}^{2}}{\operatorname{Var}(X)}\right]\\\sigma_{\hat{\beta}_2}^{2}=\frac{\sigma_{u}^{2}}{n \operatorname{Var}(X)} $$ Though we will never really know $\sigma_{u}$, the formulae provide the intuition how the variance of coefficients are determined.

In the visual example of last section, we have seen that the larger $\sigma_{u}$ causes larger $\sigma_{\hat{\beta}_1}^{2}$ and $\sigma_{\hat{\beta}_2}^{2}$, here the formulae also present the relation.

And there are two ways to contract the variance of $b_1$ and $b_2$, one is increasing sample size $n$, the other is increasing $\operatorname{Var}(X)$.

In practice, we substitute $\sigma_{u}^{2}$ by its unbiased estimator $$ s_{u}^{2}=\frac{n}{n-2} \operatorname{Var}(e) $$

where $\operatorname{Var}(\epsilon)$ is the sample variance of residuals. The term $\frac{n}{n-2}$ is for upscaling variance, because generally residuals are smaller than disturbance term, this is determined by its mathematical nature. You will get a very clear linear algebraic view in advanced econometric theory.

After plug-in we get the standard error of $\hat{\beta}_1$ and $\hat{\beta}_2$ $$ \text { s.e. }\left(\hat{\beta}_1\right)=\sqrt{\frac{s_{u}^{2}}{n}\left[1+\frac{\bar{X}^{2}}{\operatorname{Var}(X)}\right]}\\ \text { s.e. }\left(\hat{\beta}_2\right)=\sqrt{\frac{s_{u}^{2}}{n \operatorname{Var}(X)}} $$ The standard error is used when we are referring to the standard deviation of sampling distribution of estimates, specifically in econometric term, regression coefficients.

Hypothesis Testing and $t$-Test¶

Essentially, hypothesis testing in econometrics is the same as statistics. We propose a theory and test against data collected by experiment or sampling.

That being said, in econometrics, we mainly investigate if the linear relation between independent and dependent variables is plausible, so we rarely test a specific null hypothesis such that $\beta_2 = 2$, but rather $\beta_2 =0$. Therefore $$ H_0: \beta_1 = 0, \beta_2 =0\\ H_1: \beta_1 \neq 0, \beta_2 \neq 0 $$

Let's reproduce the house price example. We have the estimates, but we would like to investigate how reliable the results are. Reliability hinges on relativity, that means even if the absolute value of estimates are small, such as $.1$, as long as the standard error are smaller enough, such as $.01$, we can safely conclude a rejection of null hyphothesis without hesitation.

$t$ statistic of $\hat{\beta}_1$ and $\hat{\beta}_2$ are $$ \frac{\hat{\beta}_1-\beta_1^0}{s.e.(\hat{\beta}_1)}\qquad\frac{\hat{\beta}_2-\beta_2^0}{s.e.(\hat{\beta}_2)} $$ where $\beta^0$ is null hypothesis.

Generally, statistics are intuitive to interpret, it measures how many standard deviations (with known $\sigma$) or standard errors (with unknown $\sigma$) away from the null hypothesis. The further way, the stronger evidence supporting the rejection of null hypothesis.

Compute $t$-statistic as in formulae

Seems both $t$'s are far away from null hypothesis $\beta^0=0$, but how far is real far? Unless we have a quantitative criterion, the interpretations won't sound objective.

That's why we have significant level (denoted as $\alpha$) as the decisive criterion. The common levels is $5\%$ and $1\%$. If $t$ falls on the right of critical value of $t$ at $5\%$, we can conclude a rejection of null hypothesis. $$ t > t_{.05} $$ However in econometrics two-side test is more common, then rejection rules of $\alpha=.05$ are $$ t<t_{-.025} \text{ and }t > t_{.025} $$ Critical value of $t$-distribution is obtained by sp.stats.t.ppf, and the shape of $t$-distribution approximate to normal distribution as degree of freedom raise.

The two-side test $t$-statistic is

Therefore we conclude a rejections of $\beta_2=0$ ($t_{b_2}>t_{crit}$) but fail to reject $\beta_1=0$ ($t_{\hat{\beta}_1}<t_{crit}$).

The $\hat{\beta}_2=1.1010$ is interpreted that every $1$ yuan increment of disposable income is accompanied by raising $1.1010$ yuan on house price.

In contrast, $\hat{\beta}_1= -29181.16$ can't be interpreted that if every family has $0$ disposable income, the house price will be $-29181.16$, because it doesn't have any logical or economic sense. So the rule of thumb for interpretation of constant term: it's unnecessary to interpret constant coefficient if the it has no logical sense.